Unrooting the illusion of one-size-fits-all feedback in digital learning environments

The increased use of digital learning environment in education—in which reading is considered a means of learning—places greater emphasis on delivering or receiving instructional feedback and subsequent processing and implementation of that feedback. Instructional feedback allows adaptations to (learning) performances. Instructional feedback can be optional (the learner can decide for him-/herself if feedback is necessary and, if so, what type of feedback) or mandatory (the learner has no control over when and which feedback he/she will receive). Optional feedback can be the result of asking for feedback or observing others who receive feedback. The content of the instructional feedback range from verifying results to instructions about strategy use. Delivering instructional feedback—regardless of its content—is better than no feedback. Processing instructional feedback depends on the individual with his/her unique and shared individual characteristics. Besides these individual characteristics, the topic and the context of the task play essential roles. The interplay of these characteristics determines to a large extent how instructional feedback will be received, processed, and implemented with subsequent influence on (learning) performance

The anonymous reviewer:the relationship between perceived expertise and the perceptions of peer feedback in higher education

Peer feedback often has positive effects on student learning processes and outcomes. However, students may not always be honest when giving and receiving peer feedback as they are likely to be biased due to peer relations, peer characteristics and personal preferences. To alleviate these biases, anonymous peer feedback was investigated in the current research. Research suggests that the expertise of the reviewer influences the perceived usefulness of the feedback. Therefore, this research investigated the relationship between expertise and the perceptions of peer feedback in a writing assignment of 41 students in higher education with a multilevel analysis. The results show that students perceive peer feedback as more adequate when knowing the reviewer perceives him/herself to have a high level of expertise. Furthermore, the results suggest that students who received feedback from a peer who perceives their expertise as closer to the reviewee’s own perceived expertise was more willing to improve his or her own assignment

Reliability of Observational Assessment Methods for Outcome-based Assessment of Surgical Skill: Systematic Review and Meta-analyses

BACKGROUND: Reliable performance assessment is a necessary prerequisite for outcome-based assessment of surgical technical skill. Numerous observational instruments for technical skill assessment have been developed in recent years. However, methodological shortcomings of reported studies might negatively impinge on the interpretation of inter-rater reliability. OBJECTIVE: To synthesize the evidence about the inter-rater reliability of observational instruments for technical skill assessment for high-stakes decisions. DESIGN: A systematic review and meta-analysis were performed. We searched Scopus (including MEDLINE) and Pubmed, and key publications through December, 2016. This included original studies that evaluated reliability of instruments for the observational assessment of technical skills. Two reviewers independently extracted information on the primary outcome (the reliability statistic), secondary outcomes, and general information. We calculated pooled estimates using multilevel random effects meta-analyses where appropriate. RESULTS: A total of 247 documents met our inclusion criteria and provided 491 inter-rater reliability estimates. Inappropriate inter-rater reliability indices were reported for 40% of the checklists estimates, 50% of the rating scales estimates and 41% of the other types of assessment instruments estimates. Only 14 documents provided sufficient information to be included in the meta-analyses. The pooled Cohen's kappa was.78 (95% CI 0.69-0.89, p < 0.001) and pooled proportion agreement was 0.84 (95% CI 0.71-0.96, p < 0.001). A moderator analysis was performed to explore the influence of type of assessment instrument as a possible source of heterogeneity. CONCLUSIONS AND RELEVANCE: For high-stakes decisions, there was often insufficient information available on which to base conclusions. The use of suboptimal statistical methods and incomplete reporting of reliability estimates does not support the use of observational assessment instruments for technical skill for high-stakes decisions. Interpretations of inter-rater reliability should consider the reliability index and assessment instrument used. Reporting of inter-rater reliability needs to be improved by detailed descriptions of the assessment process

‘A double-edged sword. This is powerful but it could be used destructively’: Perspectives of early career education researchers on learning analytics

Learning analytics has been increasingly outlined as a powerful tool for measuring, analysing, and predicting learning experiences and behaviours. The rising use of learning analytics means that many educational researchers now require new ranges of technical analytical skills to contribute to an increasingly data-heavy field. However, it has been argued that educational data scientists are a ‘scarce breed’ (Buckingham Shum et al., 2013) and that more resources are needed to support the next generation of early career researchers in the education field. At the same time, little is known about how early career education researchers feel towards learning analytics and whether it is important to their current and future research practices. Using a thematic analysis of a participatory learning analytics workshop discussions with 25 early career education researchers, we outline in this article their ambitions, challenges and anxieties towards learning analytics. In doing so, we have provided a roadmap for how the learning analytics field might evolve and practical implications for supporting early career researchers’ development

Scaffolding expository history text reading:Effects on adolescents' comprehension, self-regulation, and motivation

Reading comprehension is an important predictor for academic success, yet many adolescents in secondary education face difficulties when reading their textbooks. In this quasi-experimental study, we developed a digital learning environment to scaffold students' expository text reading in seventh-grade history classrooms. Students in the experimental condition could use hints comprised of cognitive and metacognitive reading strategy instruction, whereas students in the control condition received no additional support. A comparison of posttest comprehension between conditions showed no significant differences. However, students in the experimental condition who accessed hints during history text reading performed significantly better on the posttest than students who did not use hints at all. We found no differences between conditions regarding students' self-regulated learning or motivation, but students' awareness of problem-solving reading strategies significantly increased in the experimental condition. Finally, a comparison of students with different reading levels showed that below-average readers benefitted most from digital reading practice